Some prior work has shown that LLMs perform well in NLG evaluation for different tasks. However, we discover that LLMs seem to confuse different evaluation criteria, which reduces their reliability. For further verification, we first consider avoiding issues of inconsistent conceptualization and vague expression in existing NLG quality criteria themselves. So we summarize a clear hierarchical classification system for 11 common aspects with corresponding different criteria from previous studies involved. Inspired by behavioral testing, we elaborately design 18 types of aspect-targeted perturbation attacks for fine-grained analysis of the evaluation behaviors of different LLMs. We also conduct human annotations beyond the guidance of the classification system to validate the impact of the perturbations. Our experimental results reveal confusion issues inherent in LLMs, as well as other noteworthy phenomena, and necessitate further research and improvements for LLM-based evaluation.

However, the previous methods, including fixed Simultaneous machine translation (SiMT) (Gu and adaptive policies, lack evaluation before taking et al., 2017; Ma et al., 2019; Arivazhagan et al., the next action. For fixed policy (Ma et al., 2019; 2019; Ma et al., 2020; Zhang and Feng, 2021b, Elbayad et al., 2020; Zhang et al., 2021; Zhang 2022d) starts translation before reading the whole and Feng, 2021c), the model generates translation source sentence. It seeks to achieve good latencyquality according to the predefined translation rules. Although tradeoffs and is suitable for various scenarios it only relies on simple training methods, with different latency tolerances. Compared to it cannot make full use of the context to decide an full-sentence machine translation, SiMT is more appropriate translation policy. For adaptive policy challenging because it lacks partial source content (Gu et al., 2017; Arivazhagan et al., 2019; Ma in translation and needs to decide on translation et al., 2020; Zhang et al., 2022), the model can policy additionally.